gitlab-com-seaweedfs/seaweedfs
1
0
Fork 0
mirror of https://github.com/seaweedfs/seaweedfs.git synced 2024-01-19 02:48:24 +00:00
Code Issues Projects Releases Wiki Activity
5340 commits 28 branches 278 tags 135 MiB
Commit graph

3392 commits

Author SHA1 Message Date
Chris Lu dd9f3a0104 add sleep between upload retries 2021-02-17 20:55:11 -08:00
Chris Lu aa17311063 s3: set filename when uploading file chunks 2021-02-17 20:54:53 -08:00
Chris Lu 3f8b0da677 filer: do not print password on error 
fix https://github.com/chrislusf/seaweedfs/issues/1809
 2021-02-17 02:13:52 -08:00
Chris Lu cd866664a8 skip JWT if fileId is empty 
related to https://github.com/chrislusf/seaweedfs/issues/1808
 2021-02-16 15:39:12 -08:00
Chris Lu 632c94d438 Merge branch 'extend_to_disk_type' 2021-02-16 11:06:13 -08:00
Chris Lu 0611233f16 adjust printing 2021-02-16 10:55:30 -08:00
Chris Lu 68775d29e3 fix tests 2021-02-16 10:51:03 -08:00
Chris Lu 7403cd43c5 only use "" for hdd to avoid two values for the same thing 2021-02-16 10:50:42 -08:00
Chris Lu b314d78e97 fix print 2021-02-16 10:48:28 -08:00
Chris Lu 53ca7e66ef avoid dead lock 2021-02-16 10:48:16 -08:00
Chris Lu 38efc6f572 simplify 2021-02-16 10:47:32 -08:00
Chris Lu ad70107eb4 ec balance among nodes with HDD 2021-02-16 06:22:50 -08:00
Chris Lu 3097b9a9b7 fix existence checking 2021-02-16 05:59:43 -08:00
Chris Lu ebf320ec22 avoid divided by zero 2021-02-16 05:59:24 -08:00
Chris Lu a0c6db361c avoid nil 2021-02-16 05:33:38 -08:00
Chris Lu 36f95e50a9 avoid possible nil disk info 2021-02-16 05:13:48 -08:00
Chris Lu 43101ccea0 move to the empty nodes first 2021-02-16 04:27:16 -08:00
Chris Lu 38bbef7ec1 avoid nil 2021-02-16 04:16:46 -08:00
Chris Lu cb9cc29518 volume.list display; fix updating maxVolumeCount for disk 2021-02-16 03:55:24 -08:00
Chris Lu 3fe628f04e use hdd instead of empty string 2021-02-16 03:03:00 -08:00
Chris Lu f8446b42ab this can compile now!!! 2021-02-16 02:47:02 -08:00
Chris Lu b9b5b932c5 filer: postgres2 mysql2 avoid repeatedly creating tables 2021-02-15 16:19:24 -08:00
Chris Lu 71f0c19515 2.26 2021-02-15 13:38:38 -08:00
Chris Lu 8b169aa47c s3: Critical - S3 Delete Multiple Objects, delete the bucket 
fix https://github.com/chrislusf/seaweedfs/issues/1806
 2021-02-15 13:38:29 -08:00
Chris Lu 50202d1391
Merge pull request #1805 from LazyDBA247-Anyvision/master 
better postgresql connection pool management
 2021-02-14 22:26:29 -08:00
Chris Lu 8726ef34c1 2.25 2021-02-14 22:23:43 -08:00
LazyDBA247-Anyvision 7f458d5e78 better postgres connection pool management 
adding SetConnMaxLifetime configuration (https://golang.org/pkg/database/sql/#DB.SetConnMaxLifetime)
to enable refresh of stale connections.
 2021-02-15 07:45:09 +02:00
Chris Lu 0bc3a1f9e8 disk type only supports hdd and ssd, not ready for random tags yet 2021-02-14 11:38:43 -08:00
LazyDBA247-Anyvision 51b4963e2e postgres2 & memsql2 
add escape (quote identifiers) for the dynamic sql
so tables (collections) with special characters will work.
 2021-02-14 13:14:36 +02:00
Chris Lu 876dbe3d26 collect disk types 2021-02-13 23:25:16 -08:00
Chris Lu 861f8fab9b refactoring 2021-02-13 22:34:12 -08:00
Chris Lu 881f05f5a0 adjust help message 2021-02-13 16:13:52 -08:00
Chris Lu ef76365ec2 adjust help message 2021-02-13 15:47:08 -08:00
Chris Lu 7ce647f27e support customizable disk type 2021-02-13 15:42:42 -08:00
Chris Lu 4bd8a692d8 disk type can be generic tags 2021-02-13 13:50:14 -08:00
Chris Lu 712b3e9e53 mount: release need to avoid nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1803
 2021-02-13 13:39:37 -08:00
Chris Lu 4ce56bac08 fix format error 2021-02-12 06:44:26 -08:00
Chris Lu 0604afcda7 mount: remove unnecessary flush on release 2021-02-12 06:01:04 -08:00
Chris Lu 7d9dc3c6a2 use fasthttp lib to read 2021-02-12 05:59:52 -08:00
Chris Lu 487e435679 adjust http max idle connections per host 
related to https://github.com/chrislusf/seaweedfs/issues/1802
 2021-02-12 03:47:15 -08:00
Chris Lu 0f426ce34d
Merge pull request #1801 from kmlebedev/recoveringRabbitMQ 
Do reconnect to RabbitMQ
 2021-02-11 01:05:03 -08:00
Konstantin Lebedev 94eac4f00e Do reconnect to RabbitMQ 
https://github.com/google/go-cloud/issues/2958
https://github.com/chrislusf/seaweedfs/issues/1773
 2021-02-11 13:59:36 +05:00
Chris Lu 885ca34748 volume: fail fast if idx files are missing 
fix https://github.com/chrislusf/seaweedfs/issues/1796
 2021-02-11 00:44:40 -08:00
Chris Lu a0e84c4fbc go fmt 2021-02-10 23:41:05 -08:00
Chris Lu c959aaa19b mount: load directory based on correct filer path 
fix https://github.com/chrislusf/seaweedfs/issues/1799
 2021-02-10 23:14:18 -08:00
Konstantin Lebedev 0cfed8c3cb restart replication if RabbitMQ connection closed 
https://github.com/google/go-cloud/issues/2958
 2021-02-10 15:29:05 +05:00
Chris Lu 770393a48c volume: add capability to change disk type when moving a volume 2021-02-09 23:58:08 -08:00
Chris Lu 821c46edf1 Merge branch 'master' into support_ssd_volume 2021-02-09 11:37:07 -08:00
Chris Lu a833021132 fix refactoring left over 2021-02-07 23:03:03 -08:00
Chris Lu 5e4b5109dd 2.24 2021-02-07 22:29:43 -08:00
Chris Lu 15c60cbb26 close the grpc connection after 10 hours 
related to https://github.com/chrislusf/seaweedfs/issues/1782
 2021-02-07 03:50:01 -08:00
bingoohuang 7256902fb0 fix typo offset.ToAcutalOffset to offset.ToActualOffset 2021-02-07 12:11:51 +08:00
bingoohuang 94ea3bd3a5 renaming NeedleMapType to NeedleMapKind 2021-02-07 09:00:03 +08:00
Chris Lu c3af72d950 fier store: fix elastic search regression 
fix https://github.com/chrislusf/seaweedfs/issues/1774
 2021-02-04 01:30:14 -08:00
Konstantin Lebedev 2fb4c60540 fix log warn 2021-02-03 21:34:32 +05:00
Konstantin Lebedev 24ec17219b RabbitMQ delay retry with Dead Letter Exchange 
https://github.com/chrislusf/seaweedfs/issues/1773

https://github.com/google/go-cloud/issues/2952
 2021-02-03 21:32:30 +05:00
Chris Lu a331bbb3ae filer: should return 204 on DELETE to nonexistent file 
related to

https://github.com/chrislusf/seaweedfs/issues/1776
https://github.com/chrislusf/seaweedfs/issues/1160
 2021-02-03 00:40:31 -08:00
Chris Lu 5353e38469 S3: should return 204 on DELETE to nonexistent file 
fix https://github.com/chrislusf/seaweedfs/issues/1776
 2021-02-03 00:35:44 -08:00
Chris Lu 2396ac234c filer: fix elastic search pagination 
possible fix for https://github.com/chrislusf/seaweedfs/issues/1774
 2021-02-02 11:34:12 -08:00
Chris Lu 609daaf387 s3: DeleteMultipleObjectsHandler clean up leftover empty folders 
fix https://github.com/chrislusf/seaweedfs/issues/1772
 2021-02-01 10:49:17 -08:00
Chris Lu d67ccb66c0 webdav: can start together with "weed server" or "weed filer" 2021-01-31 22:16:52 -08:00
Chris Lu 76430790b9 webdav: cache to version specific folder 2021-01-31 22:02:03 -08:00
Chris Lu 318a3d2efc 2.23 2021-01-31 20:18:06 -08:00
Chris Lu 4be51c0701 filer: leveldb and hbase may miss files when listing large directories more than 1024 
fix https://github.com/chrislusf/seaweedfs/issues/1768
 2021-01-31 20:11:44 -08:00
Chris Lu 1102ae32c4 fix concurrent map reads 2021-01-31 18:26:26 -08:00
Chris Lu 314dc1c957 filer: etcd store fix listing 
fix https://github.com/chrislusf/seaweedfs/issues/1767
 2021-01-30 13:38:44 -08:00
Chris Lu d475c89fcc go fmt 2021-01-28 15:23:46 -08:00
Chris Lu 19295600f9 mount: change option name to volumeServerAccess, with publicUrl and filerProxy modes 2021-01-28 15:23:16 -08:00
Chris Lu 990fa69bfe add back AdjustedUrl() related code 2021-01-28 14:36:29 -08:00
Chris Lu 9292796ec2 fmt 2021-01-28 14:29:14 -08:00
Chris Lu c2bf1a88ac delete from the deepest directory first when checking empty folders 2021-01-28 14:28:40 -08:00
Chris Lu e9d8201925 s3: batch purge empty folders 2021-01-28 13:20:06 -08:00
Chris Lu cf252fc0cd mount: report error when Flush() 
The error in Release() is not working.

See https://github.com/jaderhs/libfuse/blob/master/FAQ

related to https://github.com/chrislusf/seaweedfs/issues/1765
 2021-01-28 04:46:37 -08:00
Chris Lu 9a06c35da4 replicate: incremental sink only contains new and updated files 
address da08402ba2
 2021-01-28 02:39:22 -08:00
Chris Lu da08402ba2 replicate: use creation time for local incremental file sink 
related to https://github.com/chrislusf/seaweedfs/pull/1762
 2021-01-28 02:17:41 -08:00
Chris Lu 822f1ade9d
Merge pull request #1762 from kmlebedev/backupsink 
replication to create time date directory
 2021-01-28 02:03:33 -08:00
Konstantin Lebedev be1062b7fc rename in scaffold 2021-01-28 14:59:20 +05:00
Konstantin Lebedev 02fdc0a333 rename backup to local_incremental and use mtime 2021-01-28 14:56:13 +05:00
Chris Lu 5f72c388bc
Merge pull request #1764 from danlsgiga/handle-umask 
Use provided umask on mount
 2021-01-28 01:09:20 -08:00
Daniel Santos d38613c49a match upstream 2021-01-27 20:56:41 -07:00
Chris Lu b81956bcb5 mount: invalidate kernel cache when mounted to a filer path 
fix https://github.com/chrislusf/seaweedfs/issues/1752#issuecomment-768178422
 2021-01-27 10:28:37 -08:00
Konstantin Lebedev 6b54ff9912 replication to create time date directory 2021-01-27 15:01:33 +05:00
Chris Lu f20ec82a28 update help message 
see https://github.com/chrislusf/seaweedfs/issues/1758#issuecomment-768123224
 2021-01-27 00:48:31 -08:00
Chris Lu 1045ab8b07 avoid error message 2021-01-27 00:21:59 -08:00
Chris Lu 0488c84bea adjust help message 2021-01-27 00:17:52 -08:00
Konstantin Lebedev 3634811408 replication to todays date directory 2021-01-27 12:45:58 +05:00
Chris Lu 5e07afb0f0
Merge pull request #1759 from kmlebedev/sink.local 
replication to local disk storage
 2021-01-26 22:31:54 -08:00
Chris Lu b3f66199db shell: volume.fix.replication adds collectionPattern with wildcard characters 
fix https://github.com/chrislusf/seaweedfs/issues/1758
 2021-01-26 22:30:39 -08:00
Daniel Santos 0c64d60fdb Fix log message with correct mode 2021-01-26 21:45:52 -07:00
Daniel Santos 0aaaa9b470 Fix var assignment 2021-01-26 21:06:08 -07:00
Daniel Santos 5a761bb7a4 Enforce umask correctly 2021-01-26 20:42:10 -07:00
Chris Lu ad2a20c8a5 notification add ack and nack 2021-01-26 11:08:44 -08:00
Konstantin Lebedev 612b7975a1 replication to local disk storage 2021-01-26 22:50:25 +05:00
Chris Lu 3a1d3d3413 mount: properly invalidate kernel node cache entry 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-26 02:50:53 -08:00
Chris Lu cc1f3907ff filer.sync: default filerProxy to true 2021-01-25 10:14:42 -08:00
Chris Lu 62ebb917e0 mount: avoid the need to specify different cache directory for different mount 
fix https://github.com/seaweedfs/seaweedfs-csi-driver/issues/11
 2021-01-25 09:10:20 -08:00
Chris Lu 17f7c1c43f Merge branch 'mount_from_outside_cluster' 2021-01-24 19:03:02 -08:00
Chris Lu 00707ec00f mount: outsideContainerClusterMode proxy through filer 
Running mount outside of the cluster would not need to expose all the volume servers to outside of the cluster. The chunk read and write will go through the filer.
 2021-01-24 19:01:58 -08:00
Chris Lu 9bcb28a3ea avoid possible nil reader 
fix https://github.com/chrislusf/seaweedfs/issues/1754
 2021-01-24 17:19:35 -08:00
Chris Lu 2c5eac5705 2.22 2021-01-24 14:06:09 -08:00
Chris Lu 6ca10725b8 Revert "mount: when outside cluster network, use filer as proxy to access volume servers" 
This reverts commit 096e088d7b.
 2021-01-24 03:15:19 -08:00
Chris Lu 096e088d7b mount: when outside cluster network, use filer as proxy to access volume servers 2021-01-24 01:41:38 -08:00
Chris Lu ccbdb38c89 s3: fix for listing objects if more than 1000 in the folder 
many existing tools have max key set to 1000
 2021-01-24 00:27:52 -08:00
Chris Lu 80b8692688 filer.sync: replicate outside of either cluster, only need to see filers 2021-01-24 00:01:44 -08:00
Chris Lu f8dbb03bdd filer: append operation returns final file size 2021-01-23 04:19:49 -08:00
Chris Lu 759482e2ac filer: proxy all http methods to volume servers 2021-01-23 03:52:38 -08:00
Chris Lu 937cfacc01 filer: add "proxyToFileId" to reverse proxy to a volume server 2021-01-23 03:43:48 -08:00
Chris Lu 20ef3bb8d4 mount: use direct_io to avoid OS page cache 
fix https://github.com/chrislusf/seaweedfs/issues/1752
 2021-01-22 22:39:46 -08:00
Chris Lu 711c3f3939 filer: fix http status 206 setting 
fix https://github.com/chrislusf/seaweedfs/issues/1753
 2021-01-22 00:26:30 -08:00
Chris Lu 024c186a48 filer.meta.tail: output json format for easier parsing 2021-01-21 22:39:25 -08:00
Chris Lu 84f05787f8 mount: limit background requests and congestion threshold 2021-01-20 23:30:14 -08:00
Chris Lu f98817cfe6 filer: support appending to a file 2021-01-20 13:40:32 -08:00
Chris Lu 1b8e3da295 refactoring 2021-01-20 12:59:54 -08:00
Chris Lu 1a3f859c58 filer store: postgres configuration adjust text 2021-01-19 18:12:07 -08:00
Chris Lu b434f7e4e0 filer store: postgres adjust default config, add optioanl schema 2021-01-19 18:10:36 -08:00
Chris Lu d5add83e85 filer store: add postgres2 2021-01-19 18:07:29 -08:00
Chris Lu 52a8f1470e filer store: add mysql2 2021-01-19 17:21:50 -08:00
Chris Lu 93b3adba98 fix bucket creation 2021-01-19 15:55:51 -08:00
Chris Lu fa0c8d5283 fix error 2021-01-19 14:05:48 -08:00
Chris Lu 4c5b752b04 restructuring sql stores 2021-01-19 13:53:16 -08:00
Chris Lu 96354208c5 scaffold: add YugabyteDB 2021-01-19 12:35:14 -08:00
Chris Lu ca8f793978 refactor: prepare for bucket specific tables 2021-01-19 12:34:58 -08:00
Chris Lu 90ce1eec19 add more input params 2021-01-19 11:32:55 -08:00
Chris Lu f13c082583 add memsql to comments 2021-01-19 11:31:55 -08:00
Chris Lu e439b65e38 fix test 2021-01-18 01:48:00 -08:00
Chris Lu c7197470fd 2.21 2021-01-18 01:25:04 -08:00
Chris Lu 5a3386e39b Revert "Revert "mount: fake support for socket/block/character/fifo devices"" 
This reverts commit 3d5cb7eb86.
 2021-01-18 01:15:07 -08:00
Chris Lu 67faa56920 Revert "Revert "mount: add more entry type"" 
This reverts commit 4085b79d38.
 2021-01-18 01:14:58 -08:00
Chris Lu 7119e7b491 mount: ensure reading from dirty pages 2021-01-18 01:14:42 -08:00
Chris Lu de876c795d minor fix 2021-01-18 01:14:27 -08:00
Chris Lu 4085b79d38 Revert "mount: add more entry type" 
This reverts commit d7d907be27.
 2021-01-18 00:19:19 -08:00
Chris Lu 3d5cb7eb86 Revert "mount: fake support for socket/block/character/fifo devices" 
This reverts commit 61ef2d8658.
 2021-01-18 00:18:57 -08:00
Chris Lu 61ef2d8658 mount: fake support for socket/block/character/fifo devices 
to pass pjdfstest. A distributed file system can not really support these things anyway.
 2021-01-17 23:51:10 -08:00
Chris Lu d7d907be27 mount: add more entry type 2021-01-17 23:46:39 -08:00
Chris Lu 389426bbb7 s3: listing may repeat on the edge 2021-01-17 22:59:31 -08:00
Chris Lu 09f49d1c04 refactoring 2021-01-16 19:52:15 -08:00
Chris Lu a4063a5437 add stream list directory entries 2021-01-15 23:56:24 -08:00
Chris Lu 01dc8a43ba fix rocksdb 2021-01-15 18:31:29 -08:00
Chris Lu 16ad74f477 go fmt 2021-01-14 23:11:27 -08:00
Chris Lu f002e668de change limit to int64 in case of overflow 2021-01-14 23:10:37 -08:00
Chris Lu 19e52fd414 pass along hasMore 2021-01-14 22:49:29 -08:00
Chris Lu c0bcf6f6e1 implement sql changes 2021-01-14 22:44:22 -08:00
Chris Lu 5d4568b91f implement elastic changes 2021-01-14 22:42:25 -08:00
Chris Lu 893cbc8482 implement c* changes 2021-01-14 22:38:34 -08:00
Chris Lu 5ef43b9b09 implement etcd changes 2021-01-14 22:35:56 -08:00
Chris Lu 54527f0326 implement hbase changes 2021-01-14 22:34:35 -08:00
Chris Lu b5ceffe188 implement leveldb changes 2021-01-14 22:33:05 -08:00
Chris Lu c64bfb0e2e implement mongodb changes 2021-01-14 22:28:51 -08:00
Chris Lu 34a846009d implement rocksdb changes 2021-01-14 22:24:26 -08:00
Chris Lu d2ece29fff implement redis changes 2021-01-14 22:21:31 -08:00